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The Prospects Study of Educational Growth and 
Opportunity: Implications for Policy and Practice 

Michael J. Puma, The Urban Institute 
April, 1999 



Abstract 

This paper examines results from the Prospects study related to the impact of the federal Title I program for 
disadvantaged children on their academic and socio- emotional development. The nature of the program is 
discussed, with a particular emphasis on its strengths and limitations, as is the available research evidence 
on the ability of Title I to affect schools and students. To help determine future directions for Title I, a brief 
review of educational research is presented highlighting what may or may not work to alter the largely 
disappointing results found in Prospects. The paper concludes with policy recommendations for the current 
debate about the re-authorization of Title I. 

Introduction 

T n 1965 Congress passed the Elementary and Secondary Education Act (ESEA) creating the “..cornerstone* 
of federal aid to our nation’s elementary and secondary schools” (Vanecko and Ames, 1979). The ESEA, 
which began President Johnson’s War on Poverty, was the first federal aid program specifically targeted 
at children from low-income neighborhoods, and it altered the landscape of American education by greatly 
expanding the role of the federal government (Congressional Budget Office, 1993). By far the largest program 
created under the ESEA was ‘Title I: Better Schooling for Educationally Deprived Children” that was 
intended 

“...to provide financial assistance.. .to local educational agencies serving areas with concentrations 
of children from low-income families; and 

to expand and improve their educational programs by various means ...which 

contribute particularly to meeting the special educational needs of educationally deprived children' 

{PL 89-10, Section 201). 

Today, at an appropriation level of $8 billion. Title I dominates the federal elementary and secondary education 
budget reaching over 1 1 million children annually, about two-thirds of whom are in grades 1-6 (USED, 1999). 

As we near both the 35th anniversary of Title I, and the dawn of the 21st century, we face new challenges for 
achieving an educated society including increased demands to remain competitive in a global economy, higher 
expectations for student performance, the rapid expansion of information technology, and a population of 



^Title I was renamed Chapter 1 as part of the 1981 re- authorization, but then regained its original reference as Title I in 
1994. For consistency, and ease of reading, the term Title I has been used throughout this paper. 



American schoolchildren that is becoming substantially more diversel Moreover, despite a prolonged period 
of economic growth in the U.S., about one-quarter of children under six are poor (more than twice the rate for 
adults), a statistic that has changed very little over the last 20 years (National Center for Children in Poverty, 
1998). These conditions of poverty can severely reduce access to the educational supports and experiences 
that children need to be successful in school (Natriello, McDill, and Pallas, 1987). For example, recent data 
from the National Assessment of Educational Progress (NAEP, 1998) show a substantial gap in reading 
achievement between 4th grade students who attend high- vs. low-poverty schools^ (a gap of about 38 points, 
or 3-4 grade levels). The picture for African American and Hispanic children is substantially worse -- at the 
4th grade level, for example, 69 and 64 percent respectively are reading below the basic level. Poor children 
are also twice as likely to be retained in grade, more than three times as likely to be expelled from school, and 
one-third less likely to attend college (Children’s Defense Fund, 1998). 

The Title I Program 

F or the last 35 years. Title I has sought to improve schooling for educationally-disadvantaged children 
through two distinct, and somewhat inconsistent, policy instruments. On the one hand, the program 
seeks “income equity" among financially-strapped school districts burdened by the educational needs 
of large numbers of poor children, regardless of the educational achievement of individual students. Evidence 
supports this emphasis as the “...achievement scores of all students — not just poor students - declined as the 
proportion of poor students in a school increases” (Kennedy, Birman, and Demaline, 1986). On the other 
hand. Title I includes an “educational equity” goal that seeks to “close the gap” in academic achievement, 
between low-achieving students and their classmates regardless of their family’s income. This focus fits the 
common finding that students are “...increasingly likely to fall behind grade levels as their families experienced 
longer spells of poverty” (Kennedy, Birman, and Demaline, 1986). But, low-achieving students are not 
necessarily poor with between five and nine percent of Title I students having total family incomes of over 
$50,000 (Puma, et al., 1993). 

In addition. Title I is primarily a funding stream. Beyond some broad guidelines, districts and schools have 
great flexibility to decide which schools and grades receive funds, how much they receive, the types of services 
that are provided to students, the content areas that are targeted, how teaching is done, and by whom. As a 
consequence, the ultimate success of Title I depends upon the ability of local school administrators to 
determine how best to use the additional funds to serve the needs of those children who are struggling to be 
successful in school. As David Cohen (1997) pessimistically observed. Title I is “...a funding stream (and) 
it’s not plausible to expect a funding stream.. ..to have an effect on teaching and learning.” 



Hispanic children now outnumber African American children, accounting for 15 percent of the US population under 
18, a proportion that is expected to rise to about 22 percent by 2020 (Federal Interagency Forum on Child and Family Statistics, 
1998). In contrast, the percentage of African American children is predicted to remain relatively stable through 2020; by 2050, 
African Americans and Hispanics are projected to account for almost half the U.S. population. 

^Throughout this paper “high-poverty" schools are defined as those in which 75 percent or more of the students are 
eligible for free or reduced-price school meals; alternatively, “low-poverty” schools are defined as those in which 25 percent or 
fewer students are eligible for subsidized school meals. 
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Program History: Shifting Focus From Compiiance to Exceiience 

During the first 15 years, Title I increasingly focused on tightening the rules for program accountability in 
response to charges that federal funds were being used for general aid (Martin and McClure, 1969). Federal 
regulations proliferated, sanctions were developed, and procedural requirements for fiscal compliance were 
expanded to focus funding on low-income schools and the lowest-achieving students. This attention to 
program accountability overwhelmed interest in increasing student learning as compliance with procedural 
requirements was considered sufficient to meet the educational goals of the 1965 legislation (Timar, 1994). 
In particular, concerns about regulatory compliance led to the predominate use of “pullout” programs in which 
Title I students were separated from their classmates to receive remedial instruction, typically with little if any 
coordination with the instruction taking place in their regular classrooms. 

Despite this focus on regulatory compliance, program reform did receive some attention during this time with 
the introduction of the “schoolwide” option in 1978 allowing high-poverty schools (those with 75% or more 
low-income students) to achieve overall school improvement by moving away from assistance targeted to 
individual students. Pull-out instruction has long been considered an ineffective way to deal with the problems 
of disadvantaged children (Glass and Smith, 1977; Leinhardt, Bickel, and Palley, 1982; Leinhardt and Palley, 
1982; Winfield, 1991; Winfield and Hawkins, 1993). Students taught using in-class arrangements have been 
found to receive more academic instruction than students in pull-out programs (Puma, et al., 1997), and some 
have suggested that separation from the regular classroom may remove the responsibility of educating these 
children from their regular classroom teacher (Winfield, 1986). Unfortunately, requirements for local 
matching funds in place at this time precluded almost all eligible schools from adopting the schoolwide* 
program option. 

The 1980's brought the Reagan Administration’s attempt to reduce federal regulation and devolve control to 
state and local jurisdictions with passage of the 1981 Education Consolidation and Improvement Act (ECIA). 
However, because the administrative structures and veteran personnel were well-established by this time, and 
because of concerns about whether alternative programs would pass compliance audits, the increased flexibility 
did not lead to the expected program innovation leaving pullout instruction as the dominant mode of service 
delivery. As a consequence, the decade saw frequent complaints about the poor quality of many Title I 
programs, and little evidence to demonstrate that Title I was actually achieving its educational goals. These 
concerns, heightened by the 1983 publication of The Nation at Risk (National Commission on Excellence in 
Education, 1983), moved the debate about Title I from a focus on targeting and the proper use of federal 
funds, to a focus on program excellence and student achievement. In response. Congress enacted the 1988 
Hawkins-Stafford Elementary and Secondary School Improvement Act that was intended to foster overall 
school improvement by requiring greater coordination between Title I and regular classroom instruction, 
increased focus on advanced rather than basic skills, greater parent involvement, new rules for school 
performance accountability, and increased flexibility in the use of Title I funds particularly the opportunity for 
more high-poverty schools to implement schoolwide services. 

Unfortunately, these changes were subsequently found to be an insufficient tool to improve the quality of 
classroom instruction for disadvantaged children (U.S. Department of Education, 1993). There were, however, 
a couple of bright spots: 

• Title I was Targeted at the Most Disadvantaged Students. Compared to their classmates. Title I 
participants were more likely to have a total family income under $10,000, receive public assistance, 
be non-white, live with a single parent, have poorly educated parents whose native language is not 
English, and live in urban areas with the increased likelihood of being exposed to high rates of crime, 
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violence, and drug abuse (Puma, et al., 1993). Not surprisingly, Title I students were also more likely 
to score poorly on reading and math achievement tests, receive lower grades, be retained in grade, and 
receive lower ratings from their teachers on a variety of social and behavioral factors that are related 
to school success (Puma, et al., 1993). 

• Expanded Schoolwides. Significant growth occurred in the use of the schoolwide option (Wong and 
Meyer, 1998) for which there was growing support as a way to better serve the needs of disadvantaged 
children (Pechman and Fiester, 1994; Millsap, et al., 1992; Commission on Chapter 1, 1992; Wong 
and Wang, 1994). Most were located in urban areas and in large districts, they focused on the 
elementary grades (only 10% served secondary school students), and they served relatively large 
percentages of African-American, Hispanic, and limited-English-proficient students (Schenck and 
Beckstrom, 1993). These schoolwide funds were used to: create smaller classes (Millsap, Moss, and 
Gamse, 1993); hire non-instructional staff such as counselors, social workers, and school family 
coordinators (Millsap, Moss, and Gamse, 1993); increase staff development (Millsap, et al., 1993; 
Schenck and Beckstrom, 1993); expand the use of site-based management arrangements (Stringfield, 
et al., 1997); increase opportunities for school/parent collaboration (Stringfield et al., 1997); increase 
the use of computer-assisted instruction (Schenck and Beckstrom, 1993); and, in some schools, 
provide extended school day instruction (Schenck and Beckstrom, 1993). 

Despite the desired targeting and the growth of schoolwides, the 1993 National Assessment of the Chapter 1 

Program (U.S. Department of Education, 1993) identified several continuing problems that prevented the 

program from becoming the intended force to drive broader school reform: 

• A Limited Instructional Focus. Title I instruction focused primarily on basic skills, rather than 
higher-order skills, and the added instructional time amounted to less than an hour per week (Millsap, 
Moss, and Gamse, 1993)'*. Instruction was largely focused on reading/language arts (only some 
students received math instruction, and about three percent received services in non-instructional areas 
such as counseling or health education — Puma, et al., 1997). 

• An Add-on Program Operating at the Margin. Despite attempts to reduce the dependence on pull- 
out programs, this model remained the dominant method of Title I instruction up until 1994, especially 
in the low- to moderate-poverty schools (Puma, et al., 1997). 

• Use of Instructional Aides. Although classrooms in low- and high-poverty schools had, on average, 
roughly the same adult/student ratio, high-poverty schools were more likely to use less qualified 
instructional aides, and more likely to give these aides responsibility for instructional tasks (Puma, 
etal., 1997). 

• Funds Spread Too Thin. Title I funds were broadly distributed and covered nearly all counties, 93 
percent of all public school districts, and 71 percent of all public elementary schools (Moskowitz, 
Stullich, and Deng, 1993). However, this “something for everyone approach” diluted the program’s 
impact by failing to concentrate funds on the neediest schools, for example. Title I was unavailable 
to 14 percent of the elementary schools with 50 percent or more of the students eligible for subsidized 



^In addition, most Title I students received assistance for relatively short periods of time. About half of the elementary 
grade participants received assistance for 1-2 years, and nearly all of the middle school grade students had periods of 
participation lasting from 1-2 years (Puma, et al., 1997). 
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school meals, yet services were provided in almost half of the elementary schools with less than ten 
percent poor students (Moskowitz, Stullich, and Deng, 1993). As a consequence, many low-achieving 
students could not access the services they needed - in high-poverty schools, about one third of the 
children who scored below the 35 percentile on standardized tests were left unserved by Title I 
(Moskowitz, Stullich, and Deng, 1993). 

• No Impact on Achievement, Finally, as discussed in more detail below. Title I was found to be 
ineffective at closing the learning gap between low-achieving students and their classmates. 

Because of these disappointing results. Congress in 1994 passed the $1 1 billion Improving America’s Schools 
Act (IAS A) which sought to align federal resources and policies with existing state and local school reform 
efforts to create more comprehensive solutions to improving instruction for all students. There are three 
primary themes to the new legislation: 

• Standards'based Reform. States must establish high content and performance standards for at least 
math and reading/language arts. States are also required to measure progress against these standards 
using performance assessments administered to all children sometime between grades 3 and 5, 6 and 
9, and 10 and 12. States must further establish performance accountability mechanisms that provide 
continuous feedback throughout the educational system to help improve both program implementation 
and learning outcomes for students. 

• Improved Teaching and Learning Opportunities. States must help districts and schools develop the* 
capacity to help students meet the high standards (referred to as “opportunity to learn” standards or 
strategies), and must have a strategy for professional development that prepares teachers to teach an 
accelerated, high-quality curriculum. The cut-off point for operating a schoolwide program was also 
reduced from 75 percent of the students from low-income households to 50 percent. Schools not 
eligible for schoolwide programs must minimize the use of pull-out instruction, and are encouraged 
to use strategies such as extended day (before- and after-school), extended-year, and summer programs 
to increase learning time. Finally, through the use of Title I parent compacts, parent involvement 
policies, and support for training and capacity building, the new legislation also seeks to foster and 
support school-parent partnerships. 

• Increased Local Flexibility. The last component is the encouragement of local control and flexibility 
through the use of consolidated applications and plans, new waiver authority that allows schools to 
do away with federal requirements that may impede school improvement, the ability to consolidate 
state and local administrative funds, and the combination of Title I funds with other state, district, 
private funds to create “school choice” opportunities for Title I students. 

What we know about the effect of these changes is disappointingly little. A recent report from the U.S. 
Department of Education (USED, 1999) indicates that the program has improved its targeting to the highest 
poverty schools - although these schools account for 15 percent of all schools, they now receive nearly three- 
quarters of the Title I funds (75 percent of the money goes to schools with 50% or more of the students eligible 
for subsidized school meals). 
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What Can We Expect From Title I? 



Although Title I was bom in an era when we thought it was possible for government programs to overcome 
the effects of poverty, 35 years later we have become far more realistic about what is, and what is not, possible. 
Consequently, we cannot expect Title I to be the single solution to the very difficult task of compensating for 
the substantial educational deprivations associated with child poverty. It can help certainly, but we should not 
expect this single federal program to shoulder all the burden. Four key realities make this conclusion 
exceedingly obvious: 

• A Relatively Small Program. Although currently funded at about $8 billion, this is a relatively small 
amount when compared to the total cost of elementary and secondary education. Title I accounts for 
less than three percent of total U.S. school expenditures for elementary and secondary education 
(USED, 1999). 

• Money Not Pedagogy. Title I has historically been a funding source intended to supplement local 
educational expenditures, and until very recently has had little, if any, instructional requirements. 
Moreover, an early program focus on compliance with regulations, and subsequent bureaucratic 
inertia, created a program that at least until 1994 separated students from their classmates for remedial 
instruction in basic skills for only a limited amount of time each week. 

• Something For Everyone. Prior to 1994, Title I funds were spread thinly across a very large number 
of districts and schools and rarely sufficient to meet the overwhelming needs of the participating' 
students. Changes enacted in 1994 appear to have increased the concentration of the funds on the 
poorest districts and schools, but the money is still a relatively modest increment amounting to about 
$700 per Title I student. Further, these additional funds are expected to cover an array of activities 
including supplementary instruction, professional development for teachers, the purchase of 
computers, after-school or other extended-time programs, hiring support personnel (e.g., counselors), 
and parent-school partnership programs (USED, 1999). 

• Targeting the Most Highly-Disadvantaged Students. As noted above. Title I students are both 
economically and educationally disadvantaged, and concentrated in high-poverty schools where 
student needs can overwhelm the capacity of the staff. 

Clearly, the needs are great and the funds provided by Title I cannot by themselves be expected to overcome 
the challenges faced by impoverished schools and educationally-deprived children. At most, we should expect 
Title I to provide additional funds to the most needy schools, and to serve as a catalyst to improve the quality 
of education for all children in those schools. 

Early Evidence on The Impact of Title I 

R elevant research to date is of three types: attempts to link increases in general school expenditures to 
increases in student achievement; attempts to determine if the addition of Title I funds can change 
^.instructional practice; and, attempts to link student participation in Title I services to subsequent 
improvements in academic achievement. 




6 



8 



Do Higher School Expenditures Raise Student Achievement? 

Although education spending has nearly doubled from the early 1970’s (Rothstein and Miles, 1995), evidence 
from NAEP has not shown an associated increase in student achievement (NAEP, 1998). In fact, the rate of 
learning growth has actually remained stagnant over a period of about 25 years (Barton and Cooley, 1998). 
Efforts to find a relationship between current levels of school spending and the academic achievement of 
enrolled students (the so called “production function” research) dates back more than 30 years to the seminal 
efforts of Coleman, et al. (1966), Jencks, et al. (1972), and Mosteller and Moynihan (1972), and is most 
recently associated with the work of two leading researchers. On one hand, Hanushek (1981, 1986, 1989, 
1991, 1996a, b, 1997) concludes that there is no “...strong or consistent relationship between student 
performance and school resources, at least after variations in family inputs are taken into account” (Hanushek, 
1997). On the other hand, Greenwald, Hedges, and Laine (Greenwald, Hedges and Laine, 1996; Hedges, 
Laine, and Greenwald, 1994a, b; Hedges and Greenwald, 1996) conclude that “...a broad range of school 

inputs are positively related to student outcomes, and that moderate increases in spending may be associated 

with significant increases in achievement.” This lack of consensus is, as Burtless (1996) suggests, a result of 
a poverty of reliable information. Many of the available studies are old, and suffer from serious methodological 
weaknesses. 

Probably the best explanation for a lack of consensus (offered by Hanushek) is that the amount of money that 
is spent is not as important as how it is spent. If a school has poorly trained, or inadequately supported 
teachers, and a dull curriculum, investing substantial amounts of money to raise teacher salaries, or to make 
nominal reductions in class size, will not solve the problem of low academic achievement. Increasing our* 
investment in education can only make a difference if we make the right decisions about how to use the 
additional resources. The problem, then, is not one of resource deficiency, but the inefficient allocation and 
use of the resources that are available for the education of our children. 

What is the Effect of Title I Funds on Schools? 

The key findings about the effect of Title 1 on schools are, unfortunately, related to how the program operated 
prior to the policy changes enacted by Congress in 1994. First, these data show that the program’s financial 
impact was very different in high- and low-revenue districts, especially for the poorest schools. Students in 
high-poverty schools that were fortunate enough to reside in a wealthy school district received the benefit of 
more resources than students in high-poverty schools located in poor districts even with the addition of Title 
1 and other targeted assistance (Chambers, et al., 1993). Second, researchers were generally unable to find a 
link between the addition of Title 1 funds and associated changes in the organizational capacity of schools. 
Herrington and Orland (1992), for example, found no changes in service-delivery as a result of the 
programmatic changes brought about by the 1988 Amendments. Similarly, Rowan and Guthrie (1989) 
reported no consistent relationship between instructional quality and Title 1 program characteristics. Prior to 
1994, Title 1 was, therefore, found to have only a weak influence on educational practice, and the educational 
services that students received were more related to the quality of the overall school they attended (Timar, 
1994). 

What is the Effect of Title I on Student Achievement? 

Early evaluations of Title 1 were quite discouraging finding little evidence that the program was having any 
impact on student achievement. Although there were state and local data that claimed some positive effects 
(Wargo, et al., 1970), these early studies suffered from a host of serious methodological problems (David and 
Pelavin, 1977; Thomas and Pelavin, 1976; Slavin, Stringfield, and Winfield, 1990). In response, the 1980's 
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saw a wave of Title I evaluations including the Sustaining Effects Study (SES, Carter, 1984), a reanalysis of 
SES data by Frontera (1986) and others, an independent replication of the SES (Gabriel, et al., 1985), an 
analysis of Title 1 program data (Anderson and Stonehill 1986), and analysis of other existing data (The 
Instructional Dimensions Study, SES, NAEP) by Kennedy, Birman, and Demaline (1986). The key findings 
from these various studies were that: 1) the achievement gains for Title 1 participants exceeded those for 
disadvantaged non-participants in math and in reading but only for grades 1-3; 2) achievement effects were 
best for the least disadvantaged participants (i.e.. Title 1 did not help the most challenged students); and, most 
important 3) the rate of achievement gain for Title 1 participants was about the same as that for non- 
disadvantaged students. In other words, although Title 1 helped some students do better, the program did not 
achieve its goal of “compensating” for the effects of poverty on student achievement^. 

The Prospects Study 

F rustration over a lack of evidence about the effectiveness of Title I led Congress in 1988 to mandate the 
Prospects study which was to be a “national longitudinal” assessment of the impact of Chapter 1 based 
on a comparison of the “...educational achievement of those children with significant participation in 
Chapter 1 and comparable children who did not receive Chapter 1 services.”^ The study was an enormous 
effort collecting a large amount of information from students, parents, teachers, principals, and district staff. 
The sample was both nationally representative and very large (nearly 40,000 students in an initial sample of 
365 schools), and it covered a broad grade span tracking the same students from 1991 to 1994. No other 
currently available information allows analysis of the rich relationships among student characteristics and* 
childhood experiences, family environment, and school and classroom characteristics and instructional 
processes. (Details on the study design and analysis methods can be found in Puma et al., 1997; Price, 1999). 

Overall Average Effects 

The overall findings (Puma et al., 1997; Price, 1999; Vaden-Kieman 1999; Ricciuti, 1999) indicate that Title 
I assistance was, on average, insufficient to close the gap in academic achievement between advantaged and 
disadvantaged students. Student growth in academic achievement was observed to have a “lockstep pattern” 
in which the point where students started out in school relative to their classmates is where they ended up in 
later grades. In fact, variation in student outcomes is substantially larger between students than among schools, 
and the majority of total variation in student achievement is related to differences in their initial status at the 
start of school. 

An important caveat to keep in mind is that our inability to discern a compensatory effect of Title I is not 
necessarily an indication of program failure. As with earlier Title I research, the absence of a randomized 
experiment limits our ability to determine if, in fact. Title I students would have been worse off (i.e., whether 
the gap would have actually widened over time) in the absence of the services they received. That is, we do 
not know if Title I actually helped the participating students, but may have been too weak an intervention to 



^ Some have found this conclusion to be a positive indication of the impact of Title I. That is, they assume that 
disadvantaged children would have fallen farther behind without assistance, so a finding of equal rates of growth is a finding of a 
compensatory effect of Title I. This may, in fact, be true but the evidence we have from the available studies does not allow us to 
reach this conclusion. We simply do not know what would have happened to participants if they did not receive Title i services. 
To answer this question reliably would require an experimental study with random assignment to Title I and non-Title I status for 
eligible students. 

^ Section 1462, of the 1988 Hawkins-Stafford ESEA Amendments. 
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sufficiently increase their rate of learning growth in order to close the achievement gap separating them from 
their classmates. 

These findings of no program effect, and the early gap in achievement, are supported by other recent research, 
however. In particular, Alexander and Entwisle (1994), in their analysis of data from the 13 year longitudinal 
Beginning School Study, show rather dramatically that disadvantaged students have rates of increase in 
academic achievement that are on a par with other students when they are in school, but as a result of their 
impoverished home/community environments, they lag far behind in summer growth leaving them no better 
off (or worse off) at the start of the next school year. As the authors conclude, “...lower SES youngsters start 
school already badly disadvantaged and then proceed to fall further back over the years. But they would be 
worse off still were it not for the compensatory effects of schooling.” Similarly, Phillips, Crouse, and Ralph 
(1998) find that African American children start school with substantially lower math and reading skills and 
that this initial gap in achievement widens by the end of the 12th grade -- in fact, about half of the 12th grade 
Black-white achievement score difference can be explained by differences that existed when the children 
entered school. Moreover, neither socioeconomic differences, nor differences in measurable school 
characteristics, are sufficient to explain why African American students learn less over time than white students 
with similar initial levels of achievement. 

Do Schoolwides Do Better? 

As previously discussed, the last ten years has seen increased interest in the use of Title I to help disadvantaged 
children by allowing funds to be used to improve the overall environment of the school. Although these' 
schoolwide programs were allowed under Title 1 since 1978, they were only rarely implemented until the 1988 
Hawkins-Stafford Amendments removed the requirement that districts provide matching funds. More recently, 
the 1994 Improving America’s Schools Act greatly expanded this Title I option by lowering school eligibility 
from 75 percent of the enrolled students eligible for federal subsidized school meals to 50 percent. In many 
major urban school districts, this change allows essentially all Title I schools to implement a schoolwide 
program. According to Wong and Meyer (1998) the number of schoolwide programs grew from 1,200 in 1991 
(out of about 12,000 eligible schools) to over 8,000 in 1996 (nearly half of the 16,800 eligible schools). 

Despite their growth, there is scant information available on the impact of this program option on student 
achievement, and most of what is available comes from Title I program data (Pechman and Fiester, 1994; 
Wong and Meyer, 1998). The only national studies that include such information are Prospects (Puma, et al., 
1997), and the Special Strategies studies that conducted an in-depth examination of six schoolwide programs 
(Stringfield, et al., 1997). Data from Prospects show that students in high-poverty Title I schools that chose 
to adopt the schoolwide option, did not demonstrate a rate of achievement gain in reading or math that was 
higher than Title I students receiving targeted assistance (unpublished analysis). Stringfield, et al. (1997) also 
failed to find the expected positive effect of schoolwides on student achievement. These separate results do 
not necessarily mean that schoolwide programs, as implemented prior to 1994, were less effective than targeted 
Title I assistance. Rather, it is more likely that the types of changes made by the average schoolwide program 
were insufficient to make a significant difference in the achievement gains of the participating students. In 
other words, what seems to matter is not whether a school selects to serve students using targeted assistance 
or through more “schoolwide” changes, but the types of changes that are made to the instruction that students 
receive and how these changes are implemented. 
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Are There Some Successful Programs? 



The general findings on the impact of Title I on student achievement prior to 1994 notwithstanding, some 
researchers have reported that there may be some programs for disadvantaged students that can effectively raise 
their academic achievement relative to that of their more advantaged peers. An analysis of ten special Title 
I programs by Stringfield, et al. ( 1997) found that students in only two of the examined programs exhibited 
higher than average rates of achievement growth - Success for All and the Comer Development Program. 
Both programs focus on whole school reform, rather than on targeted assistance for selected students, and both 
concentrate their efforts on the early grades rather than spreading their efforts across the complete span of 
grade levels. Given the limitations of this study acknowledged by the authors (primarily the very small sample 
sizes), these results are at a minimum suggestive of program success. 

Information supporting this conclusion can also be found in the Prospects study which suggests that there are 
certain characteristics that distinguish “high-performing” high-poverty schools from other poor schools. These 
include, a balanced emphasis on both remedial and higher-order skill instruction, the use of ability tracking, 
parent and teacher support for the school’s mission that fosters a positive school climate, a more experienced 
principal, lower rates of staff turnover, and lower levels of student disciplinary actions. Combined with the 
above findings, there is evidence, therefore, to indicate that despite the failure to find an overall program effect, 
there may be innovative, high-quality Title I programs that were capable of moving disadvantaged students 
toward the goal of closing the achievement gap. To clearly establish this conclusion, however, requires 
considerably more research than is currently available, preferably using randomized experiments’. Moreover, 
we need a far greater understanding of what it is about these successful programs that may be driving the* 
higher gains in student achievement. 

Information on the Post-1994 Program 

Unfortunately, there is no direct information available on whether these pre-1994 findings have been altered 
as a consequence of the changes brought about by Congressional passage of the lASA. The only information 
on student effects that are available are derived from NAEP which tracks changes in academic achievement 
for national and state samples of all students at selected grade levels (USED, 1999). These data show that 
elementary-grade students in high-poverty schools — the two primary targets of Title I — have achieved 
significant gains in reading and math relative to the national average between 1992 and 1996. And, the gap 
between high- and low-poverty schools has been narrowed, although the differences remain substantial. Of 
course, many factors apart from Title I contribute to these gains, as they did to the losses seen during the late 
1980's and early 1990's, so these data are of only limited use in trying to understand what has happened to Title 
I after the 1994 re-authorization. 

Do We Know What Works? 

T here is increasing pressure from all sectors of society to improve the academic performance of 
America’s school children, particularly for the most disadvantaged students. There is also no shortage 
of ideas about what schools should do. But, fiscal realities require that consideration be given to both 
“what works best,” and to “what options give us the biggest bang for the buck.” That is, we need 
improvements, but as we have seen there is no compelling evidence that simply spending more money will 



^ Experiments are the only way to both deal with the myriad of complex factors that can affect student achievement, 
and to reliably attribute (barring failures in implementation) changes in student outcomes to a particular program intervention. 
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increase student achievement. 



Policymakers, and researchers, have approached the problem of finding how to improve school performance 
from at least four different perspectives. The oldest and most well researched, focuses on individual teachers 
and classrooms and includes an enormous body of work on curriculum content, instructional methods, and 
classroom practices. The second takes a broader perspective and seeks ways to improve entire schools^ 
including changes to school governance, school climate and culture, the adoption of schoolwide approaches 
to curriculum and instruction, and changes to school policies (e.g., the assignment of teachers to classrooms, 
the grouping and tracking of students, the use of various tutoring programs, and policies for assessment, 
grading, attendance, and discipline). The third, and newer perspective, focuses on systemic school reform 
that seeks to change both the individual components, and the linkages among the parts, of the educational 
system. A final perspective, broadens the perspective even farther to include everything else that affects how 
students learn, i.e., schooUfamily/community factors. The following discussion uses these different 
perspectives of educational change to examine what we know about the effectiveness of particular strategies 
to improve student achievement. It is not intended to be a comprehensive review of the available literature 
(which would take far more time and effort than this modest paper), nor a formal attempt to statistically link 
educational inputs and policy options to student outcomes. Instead, these observations are offered to place the 
limited evidence about the effect of Title I on student achievement into a broader context of what we know 
about how to improve educational outcomes, especially for low-achieving students. 

A Focus on Individual Teachers and Classroom. 

Instructional Practice. Conventional instruction, especially that targeted at low-achieving students, has been 
based on a hierarchical model of instruction which assumed that basic skills (e.g., phonetic decoding) had to 
be firmly in place before higher-order skills could be taught (Allington and McGill-Franzen, 1989). This 
traditional approach has been recently challenged by research findings suggesting that instruction for 
disadvantaged children that emphasizes reasoning and problem-solving is more effective at teaching advanced 
skills, at least as effective at teaching basic skills, and better at engaging students in learning (Rutter, et al., 
1979; Mortimer, et al., 1988; Stringfield and Teddlie, 1988; Knapp and Shields, 1990; Knapp and Turnbull, 
1990; Knapp, Shields, and Turnbull, 1992). 

Research reviews by Brophy (1986, 1989), Good and Brophy (1986), Hawley and Rosenholtz (1984). and 
Wang, Haertel, and Walberg (1993) have identified several common instructional factors that are reportedly 
correlated with improved student learning, including: the coverage of appropriate curriculum content; the 
amount of instructional time; student engagement and motivation tied to success; use of a structured teaching 
style; use of “interactive” teaching practices and effective questioning; efficient classroom management 
techniques; and the amount and quality of teacher-student social interaction (e.g., modeling appropriate 
behaviors, interactions that increase student’s self-esteem and school spirit). In addition, academic performance 
has been found to be associated with greater contact between teache rs and students (Braddock and McPartland 
1993; Darling-Hammond 1997; Lee, Bryk, and Smith 1993). Focusing on low-achieving students, the 
Prospects study (Puma, et al., 1997) found a significant relationship between student achievement and a 
balanced emphasis on remedial and higher-order skills, and Crawford (1989) reported an association between 
higher achievement and minimizing classroom disruptions, the use of academically challenging materials, and 
asking more “opinion” rather than simple factual questions. 

This, albeit limited, review of an enormous body of research, does seem to suggest that much can be done to 
improve student learning by focusing on “where the rubber meets the road,” at the classroom door. Classroom 
strategies that reduce non-instructional time can maximize the teaching that takes place, better student-teacher 
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academic interactions can provide teachers with the information they need (if they are properly trained, of 
course) to understand the effectiveness of their teaching and to adjust instructional strategies as necessary, 
better social interactions in the classroom can create a more orderly learning environment and provide students 
with greater opportunities to become engaged in learning, and stimulating instruction that combines a focus 
on basic skills with more challenging “higher-order” problem solving can engage students in an active learning 
process. Combined, these improvements to the everyday experiences of students can go a long way to 
increasing student achievement. How we raise teacher skills to the required level, however, is not an easy task 
and is discussed in more detail below. 

Class Size. Reducing class size has become the latest “magic bullet” to improve student learning, and is the 
focus of major federal, state, and local investments®. The results of early studies on class size were not 
especially positive (Doss and Holley, 1982; Christener 1987; Everston and Randolph, 1989), the “production 
function” work by Hanushek and Greenwald, Hedges and Laine provide conflicting results, and research 
reviews by Slavin (1989c) estimated that reducing classes to 21 or fewer students with a single teacher had 
only a small effect on student achievement. 

The most influential study of class size, the randomized experiment called the Tennessee Student/Teacher 
Achievement Ratio (STAR) project, concluded that students in small classes (13-17 students) had higher 
academic achievement than those in regular classes (22-25 students), and that the addition of a classroom aide 
had no effect on achievement (Mosteller, Light, and Sachs, 1996). Subsequent re-analysis by Krueger (1997), 
correcting for deficiencies in the initial study, supported the same basic conclusions but added the suggestion 
that the observed effect probably came from improved socialization when children started school, and that the 
small classes did not alter the rate of growth after this initial benefit. Grissmer and Flanagan (1998) report 
similar results using the NAEP data, but suggest that small class sizes have a multi-year effect on student 
performance - being in a small class has a large effect in that year but also a declining effect in succeeding 
years. The bottom line, therefore, appears to be that achieving significant class size reductions, especially in 
the early elementary grades, is likely to yield positive effects on student achievement particularly for minority 
students^. 

It is important to keep in mind, however, that this is a relatively expensive option for school reform, as many 
more teachers would have to be hired to make the required level of reductions in class size. Class size 
reduction would also raise concerns about the need for additional building space to accommodate the added 
classrooms, the potential difficulty of finding and recruiting qualified teachers (especially given concerns about 
the skills of our teaching staff), and possible implications for efforts to diversify the teaching profession. 

Whole Class vs. Small Group Instruction. In addition to changing class size, and the student-adult ratio (e.g., 
adding an aide), schools can alter how children are grouped for instruction. The two major alternatives are 
“whole-class instruction” in which students are taught together using a uniform mode of instruction, and 
“small-group instruction” in which students are divided into smaller clusters allowing teachers to better tailor 
instruction to the needs of individual students. The results of a meta-analysis by Lou, et al. (1996) indicate 



Q 

President Clinton has made reducing class size the cornerstone of his education policy at an expenditure of $12.4 
billion over a seven year period. 

^ According to Krueger ( 1997), the effect of small classes can account for 82 percent of the average black- white score 
difference in the third grade. Examining other data, Ferguson (1998) confirms this conclusion of a differential effect of small 
class size, and Grissmer, Flanagan, and Williamson (1998) speculate that reduced class sizes are the only plausible explanation 
for the narrowing of the black-white test score gap that has been found in NAEP. 
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that, on average, students placed in small groups have higher levels of academic achievement, hold more 
positive attitudes toward school, and report higher self-concept than students taught using whole class 
instruction. But, the authors also note that how the grouping is done, the teacher’s ability to handle small- 
group instruction, and the adaptation of instructional methods and materials for small-group learning are all 
very important for the gains to be achieved. Other researchers have also found that the use of cooperative 
learning involving group goals and individual accountability for learning is associated with positive gains in 
student academic achievement (Stevens and Slavin, 1995). 

Ability Grouping/T racking. Ability grouping, or tracking students by their academic ability, has come under 
increased criticism (Wheeler, 1992). Opponents argue that tracking stigmatizes students, relegates low-track 
students to work with less experienced (often less skilled) teachers, and that high-track students do not gain 
much from being grouped together. Harris (1998) also argues that tracking helps support the development of 
different group norms, sometimes in ways that negatively affect students’ motivation to work hard in school. 
Despite these concerns, this is the way that most children are being taught. Only an estimated 14 percent of 
8th graders are in mixed ability (heterogeneous) classes for math, 16 percent for English, and 20 percent for 
science and social studies; at the 10th grade, the percentages are 1 1 percent, 15 percent, 12 percent (science), 
and 18 percent (social studies) respectively (Brewer, Rees, and Argys, 1995, 1996). And, if one does not 
control for ability, minority students (African American and Hispanic) are far more likely to be in low-track 
classes than white students (Brewer, Rees, and Argys, 1995, 1996). 

Slavin’s (1990) influential review of the literature on ability grouping concluded that the effect of tracking on 
students of any ability was “essentially zero,” but this conclusion has been criticized because all of the studies* 
were done prior to 1978 (half of which were unpublished dissertations), and most used small samples. More 
recent results by Gamoran and Mare (1989) found that high school seniors in the college preparatory track 
scored slightly higher in mathematics than students in the non-college preparatory track. Brewer, Rees, and 
Argys (1995, 1996) found that being assigned to a below average group is associated with a five percentage 
point decline in math test scores; being assigned to an above average group is associated with a five percentage 
point increase', and, assignment to an average class is associated with an increase of less than two percentage 
points. The authors interpret these results to indicate that assigning all students to heterogeneous classes would 
help low-track children at the expense of high-track children who would show less gains. Ferguson’s (1998) 
review of these mixed results concludes that there is evidence of differences in instruction between high- and 
low-track classes but, these instructional differences are generally small and this explains the lack of consensus 
on the effect of ability grouping on student achievement. At this point, the research evidence on grouping and 
tracking practices is inconclusive, but the potential for causing negative consequences seem compelling enough 
to avoid looking at these types of practices as the solution to low student achievement. 

Tutoring. A variety of school interventions have been developed (and some have been widely implemented) 
to provide intensive tutoring to increase the achievement of disadvantaged children, most often in reading. 
One of the most widely known models (in use in over 9,000 schools in 47 states, Stringfield, et al., 1997) is 
Reading Recovery, developed by Marie Clay in New Zealand. The program has largely been evaluated by its 
developers who report positive improvements in student achievement (Pinnell, 1989; Pinnell, Deford, and 
Lyons, 1988; Deford, Pinnell, Lyons, and Place, 1990). However, these studies have been questioned by 
critics who claim that Reading Recovery improves the reading ability of some children, but probably less than 
claimed by its developers (Shannahan and Barr, 1995; Stringfield, et al., 1997). A variety of other programs 
have been developed to train and use volunteers to tutor disadvantaged children (often in reading), but research 
on these methods has also led to inconclusive results. For example, a meta-analysis of 65 studies by Natriello, 
McDill, and Pallas (1990) concluded that same- and cross-age tutoring can positively affect student 
achievement in the targeted subject area, but the recent Special Strategies study (Stringfield, et al., 1997) of 
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four popular tutoring programs'® failed to find positive effects on student achievement. These latter findings 
were, however, affected by the sample size and other methodological constraints of this study. Other research 
on the use of trained adult volunteers (Wasik, in press) concluded that “...there is a surprising lack of evidence 
about achievement effects of one-to-one tutoring by volunteers” despite the general belief that these methods 
are effective and their rapid proliferation, especially as a result of the America:Reads program. 

On balance, the research evidence is inconclusive about the general effectiveness of intensive tutoring. In light 
of these conclusions, and resource limitations, it makes most sense to focus first on improving the regular 
classroom as this is where students spend the majority of their school day. As also shown by the previously 
discussed Title I results, a modest amount of extra assistance during the child’s school day is unlikely to make 
a significant difference if the rest of his/her educational experience is characterized by poor and uninteresting 
instruction. This is not to say that special help is not needed, nor that it cannot be helpful, but that it cannot 
substitute for efforts to enhance the productivity of the entire school. 

Computer-Assisted Instruction. The growing importance of information technology in American society has 
rapidly increased the use and application of computers in American schools. Advocates of the expanded use 
of educational technology point to the potential for improving the effectiveness and efficiency of instruction 
through the use of computers, and the importance of doing so because of the broader social and business 
changes that are occurring. Others recommend caution out of fear of the possible wasteful collection of 
hardware without careful attention to its use (e.g., the image of computers in the closet), and instead place 
emphasis on the importance of software and its use by teachers (i.e., the integration of technology into 
pedagogy). 

Notwithstanding the substantial and growing use of technology in America’s schools, the research evidence 
on the effectiveness of educational technology is very limited, and what does exist tends to report positive 
findings only if technology is properly integrated into the school environment (Oppenheimer, 1997; Stringfield, 
et al., 1997). For example, a recent study of the use of computers for math instruction found that students of 
teachers who used computers for higher-order teaching in math did better on achievement tests, but students 
who used computers for “drill and practice” of basic skills did worse Wenglinsky (1998). Realizing the need 
for a greater understanding of how, and under what circumstances, technology can be used to improve student 
achievement, the President’s Committee of Advisors on Science and Technology (PCAST) issued a report 
(PCAST, 1998) recommending a broad research agenda including: fundamental work on the relationship 
between learning and education-related technologies; research on new forms of educational software especially 
instructional content and technology-enabling pedagogy; and, empirical studies to determine which approaches 
to the use of technology are most effective. In particular, the PCAST emphasized the need for large-scale and 
rigorous trials of various models of the use of educational technology to establish unequivocal evidence of 
validity or superiority. 

At this stage, the research evidence is far too limited to argue convincingly for broad investments in the use 
of educational technology as the solution to low student achievement. There are too many unanswered 
questions about how to use the new technology in schools, there is a need for better educational software tied 
to standards, curriculum, and assessments, and most important the need for professional development to ensure 
that teachers are properly trained to make effective use of computers and other technology in their classrooms. 



'^Reading Recovery (Pinnell, 1989), computer-assisted instruction, METRA (Levin, Glass, and Meister, 1984) a 
tutoring program that can be implemented by paraprofessionals or by older or same-age students, and a locally-developed 
extended day program 
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A Focus on Entire Schools 



School Climate Factors. Over the years, many researchers (McDill and Rigsby, 1973; Brookover, et al., 1979; 
Rutter, et al., 1979; MacKenzie, 1983; Edmonds 1986; and, Levine, 1990) have tried to identify “effective 
schools” as a way to determine correlates of higher levels of average student achievement. Often cited reviews 
of this literature by Purkey and Smith (1983), Good and Brophy (1986), and others have produced a variety 
of such lists, including: strong instructional leadership; a clear academic focus and high expectations for 
student learning; a dedicated and highly motivated administrative and teaching staff; an orderly and disciplined 
school environment; and, a positive school climate, particularly one that emphasizes a community spirit. The 
Prospects study (Puma, et al., 1997) identified similar school climate factors associated with “high- 
performing” schools (Ricciuti, 1999), and recent research on Catholic schools suggests that it is some of these 
same “school climate” characteristics that may allow parochial schools (which often serve high concentrations 
of disadvantaged children) to reportedly achieve higher student achievement despite lower per pupil 
expenditures (Coleman, Hoffer, and Kilgore, 1982; Coleman and Hoffer, 1987; and, Bryk, Lee, and Holland, 
1993). Finally, a recent study by Shields, et al. (1995) found that school leadership that provides a clear vision 
of success, energizes the staff, gamers necessary resources, and is able to keep the school focused on its goal 
over time is particularly important for effective school improvement. 

Much of this research has become conventional wisdom about our idea of what a “good” school should look 
like. Its appeal is probably that much of it seems to “make sense” since it conforms with our own experience 
both in school and in other types of complex organizations. But, a great deal of this literature is based on older 
(sometimes questionable) studies, and more important, just finding these types of correlations does not* 
necessarily mean that there are causal relationships between these school climate factors and student 
achievement. As a consequence, we cannot look to these organizational changes as the solution to the problem 
of how to improve student learning. As a part of a more comprehensive approach to school reform, however, 
they are certainly worth doing as they are unlikely to have negative consequences for students. 

Teacher Skills and Qualifications. There is increasing national attention on the quality of our teaching staff 
(Riley, 1999), and teachers are under increasing pressure from the drive to raise standards and toughen 
accountability, rapidly developing educational technology, and growing student diversity. At the same time, 
recent data (Lewis, et al., 1999) indicate that relatively few teachers report feeling well prepared to deal with 
any of these new challenges, and there is compelling evidence to suggest that the most academically talented 
undergraduates do not choose teaching as a career and the most talented are often the first to leave the 
profession (Mumane and Olsen, 1990). But, what do we know about teachers? 

• Teacher skills matter. Efforts by Wright, Horn, and Sanders (1997), and Sanders and Rivers (1996), 
to estimate the “value added” effect of teachers on student test scores in Tennessee have yielded three 
im portant conclusions: (1) there are certain “effective” teachers who achieve positive gains in student 
test scores, (2) the effect of teachers on test scores is cumulative over time (i.e., students in classrooms 
of very effective teachers, following an ineffective teacher, show gains but these are insufficient to 
offset prior losses), and (3) the impact of the most effective teachers is greatest for the lowest 
achieving students. Similar results have reportedly been found by researchers in Dallas and Boston 
(Haycock, 1998), and other evidence suggests that teacher quality is the single most important school 
factor affecting student achievement (Rivkin, Hanushek, and Kain, 1998). 

• Teacher verbal ability matters. Ferguson (1998) found a strong relationship in Texas between a 
teacher’s score on a basic test of literacy skills and student test scores, and also found that the effect 
of more or less skilled teachers is cumulative over time. Ferguson and Brown (1998) have found 
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similar results in Alabama. Other research tends to confirm that teachers who score higher on 
standardized exams, and those who attended more competitive schools, are more effective teachers 
(e.g., Greenwald, Hedges, and Laine, 1996; Ehrenberg and Brewer, 1994, 1995; Ferguson, 1991). 

• Teacher content knowledge is important. Goldhaber and Brewer (1996) find a strong relationship 
between student achievement and whether a teacher’s degree is in the subject area being taught. For 
example, students taught by teachers with an undergraduate or graduate degree in mathematics were 
associated with higher student achievement in math. Subsequent research by Brewer and Goldhaber 
(1996), and Goldhaber and Brewer (1996), show the same results for science. 

• Teacher supply is a constraint. Concerns about teacher quality are further exacerbated by the 
realization that in the next decade there will be an estimated need for over two million new teachers 
(Haselkom, 1997), and this may be low if the push to lower class size takes hold. Two ways to 
increase supply are to change pay scales, and to alter the requirements to enter the field. With regard 
to salaries, Mumane and Olsen (1989) found that highly compensated teachers are more likely to stay 
in teaching, but the effect is weaker for math and science teachers who have private sector alternatives. 
There is also some evidence that districts that pay higher salaries are more able to recruit higher 
quality teachers (Figlio, 1997), but other research also suggests that districts, even when able, may not 
select the highest quality teachers (e.g., Ballou and Podgursky, 1998). Although there is increasing 
interest in the use of differential compensation, we know very little about the effect of policies to 
reward the most effective teachers or those in fields where there are shortages, nor the effect of basing 
teacher pay on knowledge and skills (Odden and Kelley, 1997). With regard to changing the barriers* 
to becoming a teacher (e.g., changing licensure requirements), researchers have failed to find a 
relationship between teacher licensure and teacher quality (Ballou and Podgursky, 1998). It is also a 
concern that changing entry requirements may have adverse consequences for teacher quality and 
diversity (Villegas and Clewell, 1998; Shen, 1998). 

• Professional development is critical. In addition to attracting and retaining the best individuals, 
schools must confront the need to upgrade and maintain the skills of their existing staff. However, 
professional development has typically been ignored in most districts, and recent data indicate that 
teachers, on average, receive a day or less of training per year (Lewis, et al., 1999). There are some 
indications that high-quality training can positively affect instructional practice, but to be effective 
activities have to be both intensive and sustained (Shields, Marsh and Adelman, 1997), and must focus 
on subject matter knowledge and student learning (Cohen and Hill, forthcoming). There is relatively 
little information on the effect of professional development on student achievement. 

These results on the effect of high quality teachers on student achievement are particularly important for 

disadvantaged children who are far more likely to be taught by the most ineffective teachers (Haycock, 1998)". 

Extended School Year. Students lose a great deal of learning over the summer vacation, and this loss is 



‘ ‘ Ferguson and Brown ( 1 998) find that African American teachers who score lower on state exams in Texas were 
more likely to teach in districts with high proportions of black students. Moreover, white teachers who taught in predominantly 
black districts had lower scores than other white teachers. 
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greater for math than reading, and worse for disadvantaged students'^ (Cooper, et al., 1996). For example, 
Alexander and Entwisle (1994) show rather dramatically that disadvantaged students have rates of increase 
in academic achievement that ate on a par with other students when they are in school, but as a result of their 
impoverished home/community environments, they lag far behind in summer growth leaving them no better 
off (or worse off) at the start of the next school year. Karweit, Ricciuti, and Thompson (1994) found similar 
summer losses in student achievement using the Prospects data, and that the decline compounds over time ( 
i.e., the loss for poor children continues in each year of schooling which keeps them behind their more 
advantaged classmates despite the gains they achieve while they are in school). 

Proposed solutions to the summer drop-off include the addition of more instructional time during the summer 
months, or by more extensive revisions to the total school schedule. Under such “year-round” scheduling 
students might attend school for 45 days and then get 15 days of vacation, or attend school for 60 days and 
then get 20 days of vacation. The research on the effectiveness of such reforms has, however, been very 
limited. In a meta-analysis of the available research, Kneese (1996) found that schools that extended the 
school year achieved positive gains in student academic achievement. 

Comprehensive School Reforms. The rich literature on effective schools (see above), and a variety of studies 
of low-income schools (Venezky and Winfield, 1979; Fashola and Slavin, 1998; Stringfield, et al. 1997), have 
pointed to the need improve the achievement of all students as the best way to meet the needs of disadvantaged 
children. Such findings have led to a growing interest in the development of comprehensive or “whole-school” 
reforms, some of which are described in Exhibit I. Most recently. Congressional passage of the 1998 $150 
million Comprehensive School Reform Demonstration initiative (referred to as “Obey-Porter”) has fueled this* 
interest by providing at least $50,000 to individual schools (renewable for two more years, for a total of 
$150,000) that want to implement “proven” models of school reform (three quarters of the funds are reserved 
for Title I schools). As a consequence, implementation of many of these “packaged” reforms has spread 
rapidly despite their significant cost. For example. Comer’s SDP model which is operating in over 600 schools 
in 22 states, costs up to $278,000 per year for a 500 student school (Stringfield, et al., 1997); and Success for 
All (SFA) is operating in about 1 , 1 00 schools in 45 states at a cost of between $26 1 ,000 and $647,000 per year 
for an average school (Stringfield, et al., 1997). 

The evidence on the effectiveness of these comprehensive school reforms is either unavailable (particularly 
in the case of Paideia, Accelerated Schools, and almost all of the NAS models), or not completely convincing 
due to the use of weak research designs and the tendency of program developers to be the primary source of 
evaluation data. For example, evaluations of the Comer model by its developers at Yale University have 
involved simple comparisons of Comer students with other students in the same district. These studies have 
reportedly shown significant gains in reading and mathematics (Haynes and Comer, 1991, 1993). Similarly, 
evaluations of SFA have almost all been completed by staff of The Johns Hopkins University using 
comparisons with students in “matched” schools in the same district. Their findings have also reportedly 
shown higher achievement for SFA students with the highest gains found for the lowest-achieving students 
(Madden, et al., 1993; Slavin, et al., 1992, 1996a, b; Slavin and Madden, 1994; Fashola and Slavin, 1998). 
Research on the effectiveness of the new NAS initiatives is the most limited. Fashola and Slavin (1998) report 
some early positive results for Slavin’s Roots and Wings program comparing scores on the Maryland School 
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Suggested explanations for the findings include differences in the opportunity to practice different academic material 
over the summer (with reading practice more available than math practice), and the greater likelihood of memory decay for fact- 
and procedure-based knowledge than for conceptual knowledge. 
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Exhibit I: Comprehensive School Reform Models 



• Success For All (SFA) - developed by Slavin and 
his associates at Johns Hopkins University (Slavin, 
et al.. 1996a, b; Madden, et al., 1993), uses specific 
curricula and instructional methods to improve the 
reading ability of early grade students. A variety of 
techniques are used including preschool, extended- 
day kindergarten, one-to-one tutoring, and 
cooperative learning. 

• Comer School Development Program (SDP) - 
developed by James Comer at Yale University 
(Haynes and Comer, 1991, 1993), replaces 
traditional school organization and management 
with a collaborative school governance and 
management team, integrates social services 
especially school-based mental health, and 
enhances parent involvement. There is no defined 
curricula or instructional component. 

• Paideia - is a philosophical restructuring model 
that focuses on the use of challenging instructional 
material, didactic instruction, coaching, and weekly 
“Socratic seminars” (Adler, 1983). 

• Coalition for Essential Schools - is a broad 
school restructuring model that specifies principles 
of reform and leaves implementation to local 
school administrators and staff (Sizer, 1983, 1984). 

• Accelerated Schools - another philosophical 
approach that does not prescribe a particular 
method of instruction or curriculum but rather 
proposes a set of principles that seek ways to 
accelerate, rather than remediate, the learning of 
disadvantaged students (Levin, 1987, 1991). 

• ATLAS Communities that are based on a 
collaboration of four whole-school reformers 
(James Comer, Howard Gardner, Ted Sizer, and 
Jane Whitia) funded under the “New American 
Schools” program (as are the other six described 
below). The design features coordination among 
elementary, middle, and high school systems to 
achieve continuous experiences for students, active 
student participation in their own learning, a model 
of student as “worker” and teacher as “coach,” and 
the use of alternative forms of student assessment. 

• Audrey Cohen College System that emphasizes 
learning directed to a purpose that contributes to 
the community or the world at large. 

• Connect, developed by the technology firm of 
Bolt, Beranek, and Newman focuses on 
interdisciplinary projects that incorporate 
technology to connect students with scientific 



investigations, information, and other students. 

Expeditionary Learning/Outward Bound focuses 
on the use of learning expeditions using active 
learning, challenge, and teamwork. 

Modern Red Schoolhouse, developed by the 
Hudson Institute, emphasizes the “core curriculum” 
developed by E.D. Hirsch and makes extensive use 
of technology in instruction and assessment. 

National Alliance for Restructuring Education is 
a partnership of states, school districts, and national 
organizations that seeks to achieve the goals of 
systemic reform (see discussion below). 

Roots and Wings, developed by Slavin and his 
colleagues at The Johns Hopkins University, 
incorporates many of the elements used in Success 
for All (see above) but extends the focus to include 
mathematics, social studies, and science. 
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Performance Assessment of students in the demonstration schools to the scores of all students in the state. 
Similarly, Ross, Sanders, and Stringfield (1998) report some preliminary data showing positive increases in 
the rate of growth in student achievement for students in 25 schools in Memphis that are implementing six of 
the NAS designs along with the Accelerated Schools and Paideia models (results by model are not available, 
however). 

Walberg and Greenberg (1998a) have, however, raised serious concerns about these results stating that despite 
“...many reports of success, we find few objective evaluations conducted by independent investigators.” The 
authors fear that “conflicts of interest” can lead the developers of new educational programs to conduct their 
evaluations in ways that are likely to favor a positive outcome for their intervention. Not surprisingly, Slavin 
(1998) and his colleague Stringfield (1998) have taken great exception to this criticism arguing that SFA is 
the most rigorously evaluated school reform model. But, Pogrow (1998a,b) points out that despite Slavin’s 
claims about the success of SFA, students did not sustain their early gains leaving them almost 2.4 years below 
grade level by the end of the 5th grade. According to Pogrow, Slavin’s failure to disclose this finding is 
dangerous because he has presented SFA as the “cure” for high-poverty schools thereby misdirecting “..the 
field to overinvest in K-3 and divest itself of specialized programs..” for poor children. 

The evidence on the effectiveness of these new models of school reform, while offering some encouragement 
especially for low-achieving students, is far from conclusive as shown by the recent debates. The decision to 
commit large amounts of scarce resources to such undertakings, and the political capital needed to reach the 
level of sustained commitment needed for their success, calls for better information than we currently have 
available. Moreover, such information must be based on independent and rigorous impact evaluations.' 
Claims about the relative effectiveness of these new solutions cannot be based on simple comparisons to test 
norms, nor weak research designs such as comparisons to students in purportedly similar schools, and certainly 
not “anecdotal evidence” of success. Far too much is at stake for America’s school children to waste limited 
school resources on unproven reforms. 

Teacher Empowerment. Another popular approach to school-based reform is to increase the role of teachers 
in the governance of the school. The logic of this teacher empowerment approach is that: 1) increasing the 
decision-making role of teachers can enhance their commitment, expertise, and effectiveness; 2) which will 
then improve classroom instruction; and, 3) as a consequence, increase student achievement. And, there is 
some evidence to indicate that such “site-based management” is associated with increased teacher efforts to 
improve instruction, greater teacher responsibility for student learning, and increased teacher sharing of 
information about effective instruction (Marks and Louis, 1997). Shields, et al. (1995) further suggest that 
effective schools are places in which the administrators and the staff are actively engaged as a learning 
conununity continuously seeking ways to do a better job. But, these authors fail to find any evidence to support 
the idea that these governance changes can, in turn, have a direct effect on student achievement. Summers and 
Johnson (1995) conclude that there is “virtually no evidence that [site-based management] translates into 
improved student performance.” 

According to Drury (1998), the failure to find a significant relationship between site-based management and 
student achievement is due to: a general lack of focus on achievement by the site-based management teams; 
limits on the authority of the local decision-makers; and, deficiencies in the crucial resources available at the 
school level, namely information, knowledge, and rewards. Instead, Drury emphasizes the need to focus on 
continuous school improvement through data-driven decision making that creates “learning communities” in 
which information is constantly used to improve the effectiveness of current operations. In effect, this model 
turns schools into organizations that emphasize those characteristics of the most successful businesses. As 
Collins and Porras (1994), have found in their study of the best American companies, success comes as a result 
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of “..relentlessly asking the question ‘How can we improve ourselves to do better tomorrow than we did 
today?”’ 

A Focus on Systemic School Reform 

Standards-based Reform. Standards-based reform looks beyond the individual school to change the entire 
system of education by: (1) the development of challenging academic standards and achievement expectations 
for all students; (2) the alignment of policies and practices with these standards (including curriculum, 
assessment, professional development, instructional materials, and parental involvement); (3) the restructuring 
of governance systems to support increased academic achievement (i.e., giving schools the capacity and 
incentives to create effective strategies for preparing their students to learn the new standards); and (4) 
implementing accountability systems with appropriate incentives for improvement, and sanctions for a failure 
to reach the expected standards of performance (Smith and O’Day, 1991). The standards that are at the root 
of this process are of two types: content standards that identify what students should learn in a particular 
subject area, and performance standards which specify what students should know and be able to do 
(McLaughlin, Shepard, and O’Day 1995). 

At the same time that states, districts, and schools have begun to implement standards-based reform, some 
have questioned its underlying assumptions. First, as Weiss (1999) notes, the underlying “...theory is 
exceedingly thin, specifying overall goals, but providing little guidance on how to go about meeting those 
goals.” Second, there are legitimate concerns that disadvantaged students will not receive adequate support 
to reach the new standards (particularly because of the quality of teachers in high-poverty schools). Third,* 
such top-down reforms may be unable to alter the “core of schooling” (Elmore, 1994) because broad policy 
changes have not been found to exert a significant influence on student learning (Wang, et al., 1993). Some 
even suggest that increased centralization will create a greater focus on regulatory compliance to the detriment 
of local innovation, and sensitivity to local educational needs (Knapp, 1997). Finally, local capacity for reform 
can be an inhibiting factor (O’Day, Goertz, and Floden, 1995), as found in a study by Hannaway and McKay 
(1999) that showed that high-poverty districts face greater impediments to implementing standards-based 
reform. In particular, there is evidence to suggest that broad integrated reforms are difficult to implement 
(Zucker, et al., 1998), and that as school innovations spread from their initial “laboratory” they “lose their 
steam” (Elmore, 1994). Reforms are less likely to encounter the initial level of support, enthusiasm, and 
commitment that made them successful as a pilot program, and, later adopters may be facing significantly more 
difficult circumstances that can greatly increase the challenge of making the particular innovation work 
successfully. 

With regard to the effect of standards-based reform on student achievement, a recent study by Cohen and Hill 
(forthcoming) found that providing teachers with opportunities to learn about standards-based reform does 
affect their knowledge; when these opportunities to learn for teachers are tied to the curriculum that students 
are expected to learn, teachers change their teaching practice; and, when the assessment of student’s 
performance is consistent with the teacher training and the student curriculum, student’s achievement scores 
increase. Similar results were reported by Zucker, et al. (1998) in the evaluation of the State Systemic 
Initiatives sponsored by the National Science Foundation. The authors concluded that the provision of high 
quality professional development led to changes in classroom practice, and that those states that provided the 
“most credible evidence” of positive impacts on student achievement were those that directed their efforts at 
teachers and changing classroom practice. The results were, however, categorized as “modest” by the authors. 
In fact, Knapp’s (1997) recent review of the SSI results (Shields, Corcoran, and Zucker, 1994; Zucker, Shields, 
Adelman, and Powell, 1995; Zucker and Shields, 1995), concluded that after two years, these reforms 
“...exhibit connections between reform and classrooms that are uneven, often nonexistent, or simply hard to 
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trace.” 

Accountability Systems. Along with increasing interest in the use of standards to drive systemic reform in 
education, has come growing attention to the creation of stronger state and/or district accountability systems 
that use assessments of student performance (along with other indicators) to determine if student academic 
achievement is increasing (Fuhrman, 1999). Proponents argue that if we get schools to focus on results, and 
provide incentives for them to take responsibility for achieving the desired levels of student performance, then 
we will be able to realize the goal of higher student achievement. “Excellence focuses on the direction schools 
should be headed, and accountability identifies the parties responsible for getting there (Loveless, 1998).” 

Accountability systems are not new to education, but they have traditionally focused on inputs (not outcomes), 
and have taken a more regulatory approach (like those used in Title 1 for its entire history) in which centralized 
management creates a system of rules and punishments for non-compliance. Such regulatory models assume 
that there is a known “best” way to achieve the desired goals(s), they are costly to administer, and if poorly 
implemented can prevent the adoption of effective practices. A good example of this type of failure is the 
widespread adoption of Title 1 pullout instruction in response to tightened regulatory compliance. Because 
we are unclear about the best course to take to improve student achievement, many have argued that it makes 
more sense to give local decision makers the freedom to choose their own strategies. Moreover, education is 
a highly decentralized activity - teachers largely working alone in their classrooms - and, as a consequence, 
the idea of creating effective prescriptive regulations to achieve improvement in student outcomes is very 
unlikely. 

Examining the effectiveness of these new systems, Clotfelter and Ladd (1996) found higher pass rates in 
reading and math in the Dallas schools, where reward systems were in place for higher-performing schools, 
when compared to other schools in the state. The authors caution, however, that deciding how much of the 
observed difference to attribute to the accountability system is “hard to assess.” Similarly, studies of the 
California 8th grade writing assessment program reported improvements in what teachers did in their 
classrooms, and how students performed on the state test (Herman, 1997). But, the author acknowledged that 
assessment practices were not the sole factor that changed making it hard to disentangle the effects of testing 
from significant investments in teacher capacity building and professional development. In addition, Shepard, 
et al. (1995) examining the comprehensive performance assessment system in Maryland (which was 
implemented before standards and curriculum reforms were made), reported no student achievement gains 
attributable to the testing program in reading, and small gains in mathematics. The authors of the Maryland 
study note that, “Performance assessments - even with the diligent effort of most project teachers and the 

commitment of four university researchers -- did not automatically improve student learning When teachers’ 

beliefs and classroom practices diverge from new conceptions of instruction, it may be more effective to 
provide staff development to address those beliefs and practices directly. Performance assessments are a key 
element in instructional reform, but they are not by themselves an easy cure-all.” 

The most recent results bearing on this question come from North Carolina and Texas. These two states, 
which have arguably the most highly developed accountability systems, showed the largest average state gains 
in NAEP test scores between 1990 and 1996. Texas’ scores on its own test (TAASS) improved for the fourth 
year in a row; and North Carolina also posted significant gains on its own tests. Grissmer and Flanagan (1998) 
examined the achievement gains and concluded that the gains were, in fact, statistically significant and that 
factors commonly assumed to affect performance (spending, teacher/ pupil ratios, teachers with advanced 
degrees, teacher experience) did not explain the observed improvement in student performance. Indeed, on 
most of these characteristics, the two states are at or below the national average. The authors conclude, instead, 
that the reform policies in both states, which are very similar, are the most plausible explanation for the gains. 
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Particularly noteworthy is the finding that in Texas gains for minority students were larger; and in North 
Carolina gains for Blacks and whites were the same, although smaller for Hispanic students. Both states have 
extensive testing (in every grade) and accountability systems that reward successful schools and call low 
performing schools to task. These results are certainly not definitive evidence regarding the impact of 
accountability systems, but the findings are suggestive and encouraging. 

^^Markef^ Approaches to School Reform. A final option for systemic reform is to replace traditional 
governance structures with market mechanisms, thereby shifting power from government to parents (Chubb 
and Moe , 1990; Clune and White, 1990). Advocates of such school choice reforms believe that parents, 
having the ability to “vote with their feet,” will provide the necessary incentives to drive school improvement. 
Schools will either improve or face the loss of students, and maybe more important, those parents who are most 
likely to be involved in the school. Several models of choice have been proposed and implemented including 
magnet schools, within-district choice (e.g., choice within school clusters), charter schools, vouchers (both 
those targeted at low-income families, and vouchers open to everyone), and privatization (giving school control 
over to a profit-making company). The models differ operationally, but all seek to use parental choice to 
leverage school reform using a private sector model of producers (schools) and customers (parents). 
Privatization has the added feature of the presumed ability of the private sector to be more efficient in its use 
of resources to maximize outputs (i.e., student achievement). 

The debate over these alternatives has been particularly rancorous as many see them as a serious threat to the 
heart of American public education. To date, however, the research evidence on the effectiveness of these 
market-oriented models is far too limited to make a reasoned assessment of their relative merits. Although the* 
much debated recent evidence on the performance of charter schools in California (Walsh, 1998) indicates that 
these new schools are not being held accountable for student performance, the results of the currently ongoing 
national study of charter schools funded by the U.S. Education Department will help illuminate the 
effectiveness of charter schools. Similarly, the two major tests of vouchers in Cleveland and Milwaukee have 
not provided a clear picture of their potential impact on student performance. And, the leading examples of 
school privatization - Minneapolis, MN (Public Strategies Inc.), Wilkinsburg PA (Alternative Public Schools 
Inc.), Baltimore, MD and Hartford, CN (Education Alternatives Inc.), and the Edison Project which now 
manages nearly 50 schools - have yet to be rigorously evaluated. 

A Focus on School/Family/Community Influences 

The extent to which children learn, and achieve success in school, is obviously not related only to what 
happens to them while they are in school. Child development and learning reflect a host of influences from 
their family and community, and all their experiences before they enter school and during the time they are 
not in school. Understandably, federal resources are limited and Title I funds can only be stretched so far, 
particularly in light of the already thin distribution of the money. But, some have suggested that there may be 
ways that Title I funds can be used to reach outside of the traditional school walls to affect the academic 
achievement of disadvantaged students. In particular. Title I funds could be combined with other funding 
streams (e.g.. Head Start) to improve and expand services for poor children. This section focuses on three such 
strategies - parent involvement programs, preschool interventions, and programs for children when they are 
not in school. 

Parent Involvement. A number of the whole-school reforms discussed above, as well as Title I itself, have 
focused on the need to increase the involvement of parents in the child’s education. In fact, the 1994 changes 
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to Title I strengthened the importance of school/family community partnerships^'^ This focus on parents is 
supported by research evidence showing that parental assistance at home can have a positive effect on school 
achievement, attendance, school adaptability, and classroom behavior, and can have a positive effect on parents 
by giving them tools to help their child at home (Epstein and Hollifield, 1996). A recent review by Henderson 
and Berla (1994) shows that a number of parental activities have been found to be associated with positive 
academic outcomes for children including: establishing daily family routines (providing a quiet time and place 
to study, establishing times for going to and arising from bed, eating dinner together); monitoring out-of- 
school activities (limiting TV, arranging after-school activities and supervised care); modeling the value of 
learning and hard work; expressing high but realistic expectations for achievement (setting goals and standards, 
encouraging special talents); encouraging children’s progress in school (showing interest in school 
achievement, helping with homework, staying in touch with teachers); and, reading and discussions among 
family members (reading together, discussing the days events). 

Similarly, Phillips, Crouse, and Ralph (1998) reported that among the factors found to be the best predictors 
of the black-white test score difference was the adoption of “middle class” parenting behaviors. The authors 
also suggest that as much as half of the 12th grade difference in student achievement between white and 
African American students may be attributable to differences that existed at initial entry into school (i.e., strong 
predictors of the home environment), and Hedges and Nowell (1998) speculate that it was an increased 
emphasis on student school achievement by African American parents that largely explains the narrowing of 
the black-white test score gap observed during the 1970s. 

Obviously, schools cannot hope to alter the complex nature of parenting. But they can bring parents into the* 
educational process as partners with schools and teachers, and can do so in ways that encourage the types of 
behaviors and interactions described above. For example, D’Agostino, et al. (1998) report that Title I parent 
involvement programs that foster strong parent-teacher communications can increase parents’ efforts to work 
with their child on their education at home, and that this can subsequently increase student achievement. But, 
Epstein and Hollifield (1996) suggest that not all school/family/community partnerships lead to higher student 
achievement - it depends on how the program is structured. 

While the evidence on the impact of the home environment, especially parents’ efforts to support their child’s 
education, on academic achievement is very compelling, we know very little about how to change parenting 
behavior in ways that subsequently affect student development (see below). It is also not clear that schools 
should expect parental involvement to solve all the problems facing schools with high concentrations of poor 
children. Parents send schools the best children they have, and it is up to educators to figure out how best to 
serve their needs. At a minimum, schools, and Title I, can foster communications that create strong parent- 
teacher partnerships, and emphasize the importance of parents’ efforts to work with their child on their 
education at home. 

Preschool Interventions. Parents, and society as a whole, want children to start school ready to learn (National 
Education Goals Panel, 1994). Yet, at least prior to 1994, very few Title I dollars were actually being used 
to help prepare students for school. Instead of focusing on the “prevention” of educational disadvantages. 
Title I has traditionally focused on the “remediation” of these educational deficits, even though preschool 



The new law supports school/family/community partnerships by; requiring that parents of Title I children be 
integrated with other families; specifying that the partnerships with families should be linked to student learning; asking schools 
to develop parent “compacts” that outline how parents, school staff, and students will share responsibility for improving student 
achievement: and, allowing funds to be commingled to create unified programs that serve all parents. Funds can also be used to 
pay for transportation and child care to facilitate parents* involvement. 
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programs have been authorized under the Act since its inception. 

Over the past three decades. Federal, State, and local governments, private foundations, and private industry 
have funded a variety of programs that have sought to increase the cognitive, and often the socio-emotional, 
development of young children through two different strategies: (1) single ^^generation'' models that focus 
on either direct early education services for children, or services to the parents of young children (either to 
improve parenting knowledge and behaviors, and/or to provide adult education/job training to improve the 
material well-being of the home); and, (2) the newer ^^two-generation^^ programs that focus on the parent and 
child together (or the entire family) as a single entity. What we know about these programs is summarized 
below: 



(1) Early Education for Children 

High-quality early childhood programs consistently show large short-term effects on children's 
cognitive development (Barnett, 1995; Consortium for Longitudinal Studies, 1983; Campbell and 
Ramey 1994; Reynolds, 1992; McKey, et al., 1985; Lee, Brooks-Gunn, and Schnur, 1988; Layzer, 
Goodson, and Layzer, 1990), and some programs show positive effects on socio-emotional functioning 
(Lee, Brooks-Gunn, and Schnur, 1988; McKey, et al., 1985; Honig, Lally, and Mathieson, 1982). 

In general, these cognitive effects fade out in the early elementary school years (Barnett, 1995; 
Castro and Mastropieri, 1986; McKey, et al., 1985), especially if there are inadequate environmental 
supports during these early years of schooling (Ramey and Ramey, 1992). 

However, some high-quality child-focused interventions have longer-term benefits including, 
reduced grade retention and special education placement (Barnett, 1995; Consortium for Longitudinal 
Studies, 1983; Schweinhart, Barnes, and Weikart, 1993), as well as higher rates of high school 
graduation and employment, and lower rates of criminal behaviors and welfare dependence 
(Schweinhart, Barnes, and Weikart, 1993; Yoshikawa , 1995). Preschool programs that include early, 
intensive intervention, along with continued follow-up as children enter school, have the strongest 
effects in these areas (Wasik and Karweit, 1994; Yoshikawa, 1995; Ramey and Ramey, 1992). 

(2) Parenting Programs 

There is evidence that some parenting programs can affect parental knowledge, attitudes, and 
behavior (Johnson and Walker, 1991; Travers, Nauta, and Irwin, 1982; Andrews, et al., 1982; St. 
Pierre, et al., 1995; Quint, Polit, Bos, and Cave, 1994), but Olds and Kitzman (1993) show mixed 
results for home-visiting programs targeted at low-income families with infants. 

There is little or no evidence that parenting programs produce the hoped-for linkage between 
changed parent behaviors and improved child outcomes (Barnett, 1995; Clarke-Stewart, 1983; 
White, Taylor, and Moss, 1992; Ramey, Ramey, Gaines, and Blair, 1995; Lamer, Halpem, and 
O’Hawkey, 1992). 

(3) Adult-Focused Programs 

Adult education and training programs have not been able to greatly increase adults* literacy skills 
or job opportunities (Datta, 1992; Duffy, 1992; Mikulecky, 1992; Moore and Stavrianos, 1994), and 
even when these programs do affect the attainment of a GED this certification does not relate 
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positively to enhanced skill levels and is not the economic equivalent of a high school diploma 
(Mumane, Willett, and Parker-Boudett, 1995; Cameron and Heckman, 1993). 

• Job training and search programs have at best small effects on employment, and income (Fischer 
and Cordray, 1995) with an average effect of three to five percentage point gain in the rate of 
employment, and about a 13-19 percent increase in earnings (an average of $50 to $135 per quarter). 

• Welfare to work programs have not lifted substantial numbers of adults out of poverty (Gueron and 
Pauly, 1991). As Fischer and Cordray (1995) conclude, “If the policy goal is to end poverty or welfare 
receipt, then the interventions... have clearly failed. If, however, the goal is to increase earnings and 
decrease welfare receipt, then these programs have generally succeeded.” 

(4) Two-Generation Programs 

• Early results of two-generation programs have not been encouraging. The federal family literacy 
program. Even Start, has been linked to positive impacts on the home environment, and early 
cognitive gains for preschoolers, but these gains appear to “wash out” when the children enter school 
(StPierre, Gamse, and Alamprese, 1998). And, the best study of this approach, the recent evaluation 
of the Comprehensive Child Development Program (St. Pierre, et al., 1994), did not produce any 
significant impacts on the economic self sufficiency, or parenting skills, of participating mothers; nor, 
did it yield any significant impacts on children’s cognitive or socio-emotional development. This was 
despite the high cost of the program - an average of about $16,000 per family per year. 

In summary, there is strong evidence to indicate that high-quality, intensive preschool programs can have short- 
term positive effects on cognitive development, but there is little if any evidence to support the idea of 
positively affecting child development through the parents. There is, however, general agreement that 
preschool programs alone are not enough to ameliorate the effects of poverty (Hebbler, 1985; Karweit, 1994). 
In particular, the research indicates that the early gains made by disadvantaged children can quickly fade-out 
once the children enter school. One explanation for this fade-out effect is offered by Lee and Loeb (1995) who 
suggest that although disadvantaged students gain benefit from attending Head Start, they go on to attend 
schools that are of lower “quality” than their more advantaged peers and this tends to undermine their 
previously obtained gains. The authors conclude “...our nation’s policies that allow disadvantaged children 
to be concentrated in low-quality schools actually promote an increase in socially induced learning differentials 
as children advance through the educational system.” 

Non-school Time 

Harris (1998), in a highly discussed book, shows the importance of peer interactions for child development. 
According to Harris, it is not that “parents don’t matter,” as the popular press claimed, but that peers matter 
more than we have previously thought. As a consequence, she concludes that schools need to pay more 
attention to the social interactions that occur among students. According to Clark (1993) students spend 70 
percent of their waking hours outside of school. And, as Steinberg (1996) reports from a study of 20,000 
teenagers “....students’ time out of school is seldom spent in activities that reinforce what they are learning in 
their classes. More typically, their time and energy is focused on activities that compete with, rather than 
complement, their studies.” For example, Steinberg found that two-thirds of the teens were employed, and half 
were working more than 1 5 hours per week. 

As part of the 1993 National Assessment of Chapter 1 (Title I) conducted by the U.S. Department of 
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Education, Chimerine, Panton, and Russo (1993) examined how children use their out-of-school time and 
recommended that Title I should not supplant community-based activities for children, but should “encourage 
children to take part in productive out-of-school activities, facilitate coordination among existing programs, 
raise awareness among parents and community members. ...and help ensure that students have access to the 
programs they want and need...” Title I can also offer “..instruction before or after school and during school 
vacations... (to) reduce the amount of class time students miss for pullout programs.” But, as noted above, we 
must worry about asking Title I to do too much, thereby diluting its impact on what happens to children while 
they are in school. At most. Title I should see to foster efforts to encourage children to take part in productive 
out-of-school activities, and support community collaborations that improve access to, and the quality of, local 
out-of school programs for children. 

CONCLUSIONS 

T itle I, although an important program, cannot by itself compensate for the substantial educational 
deprivations associated with child poverty. Even at $8 billion, it is small relative to the total cost of 
U.S. elementary and secondary education. It is also primarily a funding source intended to supplement 
local educational expenditures, and as such cannot be expected to have a substantial effect on classroom 
instruction, where most of education takes place. Moreover, an early program focus on compliance with 
regulations, and subsequent bureaucratic inertia, created a program that at least until 1994 most often separated 
students from their classmates for remedial instruction in basic skills for only a limited amount of time each^ 
week. Finally, for most of its history, funds were thinly spread and rarely sufficient to meet the overwhelming 
needs of the participating students. Changes enacted in 1994 have increased the concentration of the funds 
on the poorest districts and schools, but the money is still a relatively modest increment in the available 
resources — an annual increase of about $700 per Title I student, most of whom are substantially economically 
and educationally disadvantaged. 

What Do We Know About the Impact of Title I? 

The 1994 re-authorization of Title I signaled an important shift in the program’s focus from primarily being 
a way to provide remedial instruction for poor children, to supporting state and local efforts to ensure that all 
children reach challenging academic standards. What we know about the program’s impact, however, is 
almost entirely based on information from the pre-1994 period. A recent report from the U.S. Department of 
Education indicates that the program has improved its targeting to the highest poverty schools since 1994, and 
tries to use the recent small NAEP gains as an indication of the impact of the 1994 policy changes. Of course, 
many factors apart from Title I contribute to these gains, as they did to the losses seen during the late 1980's 
and early 1990’s, so these data are of only limited use in trying to understand what has happened to Title I after 
the 1994 re-authorization. 

Does Money Alone Matter? 

Even though Title I is primarily a funding mechanism, available research does not show that simply increasing 
school expenditures will necessarily lead to better student outcomes. At least prior to 1994, Title I funds had 
only a weak influence on educational practice, and the educational services that students received were more 
likely to be related to the quality of the overall school they attended. 
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Does Title I “Close the Gap” in Student Achievement? 



Prior to the changes brought about in 1994, the available research data mirror findings that have been reported 
throughout the program’s history, i.e.. Title I serves those students who are clearly most in need of 
supplementary assistance, but the nature of the help they receive is by insufficient to close the gap in 
academic achievement between them and their more advantaged classmates. Student growth in academic 
achievement is observed to have a “lockstep pattern” in which the point where students start out in school 
relative to their classmates is where they end up in later grades. Additionally, schoolwide programs, as 
implemented prior to 1994, do not appear to be more effective at moving students toward the goal of closing 
the achievement gap. None of these findings are surprising given the previously discussed realities of the 
program’s size, scope, and implementation. 

There is, of course, variation in the extent to which Title I services are effective, and there is suggestive 
evidence that some high-quality programs are more effective at raising the performance of disadvantaged 
children. Greater research is needed to determine if these results are reliable, however, and the extent to which 
these programs are better than other types of school interventions. 

What Guidance Can We Obtain From Research? 

In light of the disappointing results, at least prior to the 1994 changes, it seems sensible to see if we can glean 
any guidance on how to improve the federal program from educational research. It is important to keep in 
mind that “...no conclusion of scientific research can be converted into an immediate rule of educational art”* 
(Dewey, 1929). Statistical data cannot be directly converted to prescriptions about what ought to be done - 
actions for improving education also require normative judgements about what we as a society want to achieve 
for our children, and what types of tradeoffs we are willing to make to obtain these goals. At best, scientific 
research can point the way, and help define choices and options for action. The following are a few of those 
signposts: 

Focus on all children. Years of Title I research shows us that, like an old house, supplementary funds and 
programs cannot be added to a “weak foundation.” No matter how good the Title I staff, nor how hard they 
work, they cannot overcome the problems of a poor school environment with less than an hour of additional 
remedial instruction per week. Alternatively, there is sufficient evidence to indicate that the best way to 
improve the learning for disadvantaged children is to improve what happens throughout their entire school day, 
and that means improving the educational environment for all children rather than just trying to target a few 
children at the margin. Only in this way can we hope to overcome the achievement “gap” faced by poor 
children at school entry. 

Comprehensive school reforms: Promise vs. Reality? Despite the growing interest in “whole-school” 
reforms, and the recent increased availability of federal funding for their implementation, we know far less than 
we think (or hope to know) about the relative strengths of many of the “packaged” models that come with great 
claims about their effectiveness. Schools are spending enormous amounts of their precious resources on 
programs that promise to raise the achievement of all students without clearly understanding if their chosen 
intervention can work in their school, or what it takes to implement these broad reform strategies. Too many 
schools get whip-sawed as they seek the latest cure-all and abandon old ideas when they fail to yield the 
promised academic gains in one or two years. Although schools need to decide how to improve what they are 
doing, the choice of a strategy must be based on reliable information derived from independent and rigorous 
impact evaluations. Claims about whether a particular intervention is “effective” cannot be based on simple 
comparisons to test norms, nor weak research designs such as comparisons to students in purportedly similar 
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schools, and certainly not “anecdotal evidence” of success. Far too much is at stake for America’s school 
children to waste limited school resources on unproven reforms. 

A focus on instruction is important Strong academic leadership, a clear academic focus combined with high- 
expectations, a dedicated and motivated staff who are willing to learn themselves, and, a positive and 
supportive school and community environment can make a difference in how students learn. In particular, it 
is critical to focus on “where the rubber meets the road,” at the classroom door. Classroom strategies that 
reduce non-instructional time will maximize the teaching that takes place, better student-teacher academic 
interactions can provide teachers with the information they need (if they are properly trained, of course) to 
understand the effectiveness of their teaching and to adjust instructional strategies as necessary, better social 
interactions in the classroom can create a more orderly learning environment and provide students with greater 
opportunities to become engaged in learning, and stimulating instruction that combines a focus on basic skills 
with more challenging “higher-order” problem solving can engage students in an active learning process. 

Although the evidence on standards-based reform is still quite thin and fragmentary, the main idea behind this 
movement is in line with a stronger body of research showing the importance of curriculum and instruction 
to student learning. Consequently, states must set clear high standards and districts and schools must be 
responsible for ensuring that all students are provided with the curriculum, teaching practices, and assistance 
they need to attain these standards. And, schools need meaningful ways to assess progress against those 
standards that should not just be used to “keep score” but to provide meaningful feedback to students, parents, 
teachers, and administrators. The entire school must become a learning community in which everyone is 
focused on the single goal of improving student learning, and data are used in a continuous way to monitor* 
progress and adjust the course when necessary. 

Teachers are the critical component. Probably the most compelling evidence we have shows that teachers 
matter. Setting high standards, and expecting all children to learn a challenging curriculum, is doomed to 
failure without the teaching staff who can effectively bring all children to the desired point of learning. This 
means finding polices to attract and retain the best individuals, eliminating the use of aides or the most 
inexperienced teachers to teach the most challenged students (or to teach in the most impoverished schools), 
and ensuring that teachers have the content knowledge and teaching skills needed to meet the demands placed 
upon them. Strong professional development should be a key ingredient of any school reform strategy, as 
should increased opportunities for professional collaboration‘s^. 

Technology can be a tool but should not be viewed as a substitute for good teaching. In fact, we know too little 
at this point about the effectiveness of computers per se for increasing student achievement, and there are 
sufficient worries about the lack of quality content-based software, and adequately trained teachers and 
instructional methods, to slow the rush to accumulate hardware in schools. An interesting idea advanced by 
Eva Baker is to see computers as the “default strategy” for instruction in particular subject areas where the 
classroom teacher may lack adequate content-area preparation. Of course, this will require high-quality 
content-based software to be successful. 

Develop and sustain the capacity to improve student outcomes. School administrators, and especially 
teachers, must have the capacity to meet the new challenges and demands facing them and the rest of society 
at the dawn of the 21st century. The movement to set high standards and expectations, and to develop 



An interesting idea suggested by Peg Goertz is to turn some teachers in every school into 11 month employees and 
to use this additional time to work on curriculum, instructional development, policy setting, and planning professional 
development. 
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accountability systems to focus attention on teacher and student performance, are certainly all important. But 
while these activities create the “will” to improve our schools, they do not address the “skill” needed to achieve 
the ambitious goals. In particular, there is ample evidence to suggest that effective school reform requires 
substantial time for implementation, continued staff buy-in, the commitment to see the program through, and 
the human and material capacity to succeed. 

Class size and instructional grouping. Smaller class size appears to be important for higher student 
achievement during the beginning years of school when children are learning their basic skills and adjusting 
to the new social setting of a school. The evidence does not seem to support broad investments in class size 
reductions, and the cost of doing so could easily overwhelm other important uses of scarce resources. 
Strategies such as small group instruction within the classroom, and cooperative learning, appear to be 
beneficial but only if properly implemented, supported by instructional materials targeted at small group 
instruction, and if teachers are well trained in the necessary techniques. 

Reform needs to be multi-dimensional. The available research evidence does not point to a single “magic 
bullet” that will by itself raise the level of student achievement, especially for the lowest-performing children. 
The focus should not, therefore, be on providing a lot of one thing, but some of many things. That is, school 
reform should include a variety of ingredients but these must be based on sound research, and focused on the 
clear goal of improving student learning. 

Non-school factors. The Prospects results, and findings from other research, provide strong and compelling 
evidence that disadvantaged children start school behind their classmates, and never “catch up” from these* 
early deficits. This suggests that increased investments in early education programs are clearly needed. But 
these programs must themselves be of high-quality and must be tied to subsequent high-quality school 
instruction for the gains to be sustained. 

Other non-school factors such as parent/school/community partnerships, out-of-school time, school 
accountability and incentive systems, and changes in local school governance, suffer from a fragmentary 
research base. We know relatively little about what works best in each of these areas, and how to achieve 
linkages between what schools are capable of doing in these areas and improvements in student academic 
achievement. 

Why don^t we know more? The state of educational research is glaringly poor. Although, there are hundreds 
of articles published every month, many by very talented scholars, most of what we know about education is 
based on at best weak research designs, and at worst, unsupported claims or anecdotal evidence. Of all the 
social policy areas, education is certainly the least well supported by sound research. This is not to say that we 
have learned nothing from the last 40 years of educational research. Although we may lack definitive answers 
about what works best, we know a great deal more about what are the right questions to ask, and where to look 
for effective solutions. 

The federal government, and national organizations, need to take a much greater leadership role in the 
generation of high quality information on “what works” in education. We need continuous experimentation 
to provide information that support data-driven decision-making in schools as learning communities. Only 
in this way can policymakers, and schools, make rational decisions about what to do to improve student 
learning. In particular, we have little, if any, information on cost-benefit tradeoffs. In a world where resources 
are limited, we need to know which school changes/reforms offer the greatest “bang for the buck.” 

We also need commitment to make reforms work and to give them a “fair” test. Too often, schools and policy 
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makers look for quick improvements and end up jumping from one new fad to another. Moreover, because 
this has become common practice, it’s hard to muster the necessary support and commitment to any program 
that is needed for success -- teachers often take the approach that “this too shall pass!” Where it does happen, 
it’s often on small demonstration projects which in most cases do not show the same results as the program 
“scales up.” 

What Should We Do to Improve Title I? 

The value and importance of education is deeply engraved in American culture — we place a high value on 
learning, and we cannot afford to waste the potential of our children if we want to maintain our economic and 
political strength well into the 21st century. But, we need to seriously re-think how we have elected to use the 
relatively scarce Title I resources and how they will be used in the future. 

1. Use Title I as a catalyst to drive standards-based reform to the classroom level Albeit limited 
evidence suggests a need to “stay the course” with standards-based reform - we need to set a clear focus on 
improved student achievement, set expectations about what we want our schools and students to achieve, and 
seek ways to move all students to higher levels of learning. Title I cannot do this alone, but it can provide an 
important source of leverage, particularly with regard to moving this effort from the states, where progress is 
being made, down to the classroom. This means focusing more on the “skill” side of the equation than on the 
“will” side, i.e., addressing the capacity of teachers and school administrators to implement higher standards 
and a more challenging curriculum. 

2. Continue to target money to the most disadvantaged schools. At the time of the last re-authorization 
of Title I, the Washington Post ran an editorial that said, ‘This is the time to improve Chapter 1, to help more 
children by helping the schools most in need. For too long. Congress has been spreading the money like 
peanut butter - thinly and evenly to most school districts. That way, the program is palatable to all. But it 
doesn’t do much good in resource-poor schools ( January 21, 1993).” Since 1994, it appears that progress has 
been made in this regard, but it needs to be sustained and deepened to concentrate the funds on the poorest 
schools to provide sufficient resources to overcome the enormous challenges faced by educationally- 
disadvantaged children. Serious consideration should also be given to focusing the activities that Title I 
programs are expected to accomplish - broadening the mandate without substantial increases in funding just 
dilutes the program’s ability to meet its primary objective. This could, for example, mean targeting the money 
on certain subjects (e.g., reading, writing, and math), on the early grades to give children a strong start toward 
continued school success, and on certain functions such as improving curriculum and instruction, and building 
school and teacher capacity (see below). 

3. Focus on instruction and teachers. It is quite clear that having a good teacher is one of the most 
critical ingredients of achieving higher student performance, especially for disadvantaged kids. This suggests 
a need to focus Title I on several aspects of building the capacity of the teaching force. This could include, 
for example, incentives to hire and retain the most qualified teachers, incentives to significantly reduce “out 
of field” teaching through either changes in hiring or professional development, the elimination of unqualified 
aides for instruction, incentives to assign the most qualified teachers to the most challenged students and 
schools, and high-quality professional development focused on content and content-related skills. 
Consideration should also be given to federal efforts (through public/private ventures) to support the 
development of technology-based content-focused instructional curricula as a “safety net” for students exposed 
to less adequately trained teachers. 

4. Build capacity. Title I now has a strong focus on accountability and high standards, but pays relatively 
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little attention to much-needed capacity building. Encouraging and supporting the development of high 
standards and challenging curriculum is important, but many schools and teachers cannot succeed at getting 
students to achieve the new expectations without the capacity to teach in new and different ways. As a 
consequence, technical assistance is needed but where should it come from? Most states and districts have 
trimmed their staff and therefore lack the resources needed to fill the gap. One option could be for the federal 
government, through Title I and other federal programs, to focus on: providing solid research-based 
information and guidance that can serve as the basis for needed technical assistance; and, encouraging (through 
a combination of leadership and incentives) the creation of local business/university/state partnerships to 
provide help and support to districts and schools. 

5. Improve accountability. Title I can, and should, serve as an educational laboratory where districts 
and schools are able to experiment with new approaches that appear to be effective. But, they must be 
required to evaluate these new ideas using rigorous evaluation methods conducted by independent evaluators. 
An excellent model is the AFDC waiver demonstration that both encouraged experimentation with new 
approaches to reducing welfare dependence, and also required the use of rigorous evaluation. Such an 
approach will spur the growth of new models, put them to the test to see if they in fact are effective, and help 
build a knowledge base that schools and policy makers throughout the country can use for future decisions. 
Further, such an accumulated base of research will eventually inform the even harder decisions about tradeoffs 
between one type of change versus another. Because educational funds will always be constrained, such 
assessments of alternative courses of action must be made, and must be based on accurate and reliable 
information about the relative costs and benefits of different educational strategies. This is particularly 
important for the ever expanding list of “comprehensive school reforms” that are proliferating without a solid* 
test of whether they are effective. 

There also needs to be a continuous monitoring system that allows for the assessment of what is happening 
under federal and state policy reforms. It is appalling that we are five years from the last Title I authorization 
and have essentially no information on what has transpired since the 1994 changes were enacted, especially 
as it affects students. In addition, the 1994 re-authorization reduced the required student assessments to 
particular grade levels but eliminated the requirement to obtain longitudinal information on student progress, 
i.e., collecting data on student achievement in adjacent grade levels. While increasing the testing burden on 
schools and students is undesirable, we need solid information on student gains to better understand how well 
we are doing. Options to meet this need should be explored and could include providing incentives to states 
to test reliable samples of students in each grade level on an annual basis, or expanding the current state-level 
NAEP initiative. 

6. Tackle the problem at school entry. It is clear that much of the challenge faced by disadvantaged 
children is in place when they start school, and many of them are unable to bridge the gap between them and 
their more advantaged classmates. Although Title I alone cannot solve this problem, it can do at least two 
things that can help: (1) focus a greater amount of the funds provided to schools on the earliest grades to do 
what can be done to put these children on firmer ground before they fall too far behind and cannot catch up, 
and (2) provide leadership and coordination among programs to create a “seamless web” of services for 
disadvantaged children from birth through early elementary school - the idea of policy and program alignment, 
and high standards, needs to be extended downward to cover the earliest years of a child’s development. 

7. School/family/community partnerships. What happens outside of school is critically important for 
how well children do in school, especially poor children who live in an impoverished home and community 
environment. The 1994 changes increased the importance of these activities, and these should be continued 
and refined where needed, but Title I cannot be expected to make major changes in these conditions. Title I 
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resources are simply insufficient to overcome the disparities in school readiness between poor and non-poor 
children, or to address other consequences of poverty. Instead, the role of Title I should be limited to indirect 
coordination and collaboration to prevent a further dilution of Title I funds. Attention to the role of families, 
communities, and “out-of-school” time should be part of a larger coordinated and “aligned” federal approach 
to meeting the needs of disadvantaged children. As part of such an effort, special attention should be given to 
the “summer loss” problem that has been repeatedly identified for disadvantaged children. 
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